3. Echopype demonstration notebook 2¶
3.1. Watching a solar eclipse using a moored echosounder¶
Jupyter notebook accompanying the manuscript:
Echopype: A Python library for interoperable and scalable processing of ocean sonar data for biological information
Authors: Wu-Jung Lee, Emilio Mayorga, Landung Setiawan, Kavin Nguyen, Imran Majeed, Valentina Staneva
3.1.1. Goals¶
Illustrate a common workflow for echosounder data conversion, calibration and use. This workflow leverages the standardization applied by echopype. and the power, ease of use and familiarity of libraries in the scientific Python ecosystem.
Demonstrate the ease to interoperate echosounder data with those from a different instrument in a single computing environment. Without
echopype, additional wrangling across more than one software systems is needed to achieve the same visualization and comparison.
3.1.2. Description¶
This notebook uses EK60 echosounder data from the U.S. Ocean Observatories Initiative (OOI) to illustrate a common workflow for data conversion, combination, calibration and analysis using echopype, as well as the data interoperability it enables. Without echopype, additional wrangling across more than one software systems is needed to achieve the same visualization and comparison.
We will use data from the OOI Oregon Offshore Cabled Shallow Profiler Mooring collected on August 20-21, 2017. This was the day before and of a solar eclipse, during which the reduced sunlight affected the regular diel vertical migration (DVM) patterns of marine life. This change was directly observed using the upward-looking echosounder mounted on this mooring platform that happened to be within the totality zone. The effect of the solar eclipse was clearly seen by aligning and comparing the echosounder observations with solar radiation data collected by the Bulk Meteorology Instrument Package located on the nearby Coastal Endurance Oregon Offshore Surface Mooring, also maintained by the OOI.
The data used are 15 .raw files with a total volume of approximately 1 GB. With echopype functionality, the raw data files hosted on the OOI Raw Data Archive (an HTTP server) are directly parsed and organized into a standardized representation following in the SONAR-netCDF4 v1.0 convention, and stored to the cloud-optimized Zarr format. The inidividual converted files are later combined into a single entity that can be easily explored and manipulated.
3.1.3. Outline¶
Establish connection with the OOI Raw Data Archive and generate list of target EK60
.rawfilesProcess the archived raw files with
echopype: convert and combine into a single quantity (anEchoDataobject) in a standardized format.Obtain solar radiation data from an OOI Thredds server.
Plot the echosounder and solar radiation data together to visualize the zooplankton response to a solar eclipse.
3.1.4. Running the notebook¶
This notebook can be run with a conda environment created using the conda environment file https://github.com/OSOceanAcoustics/echopype-examples/blob/echopype_paper/binder/environment.yml. The notebook creates a directory ./exports/ooifiles and save all generated Zarr and netCDF files there.
3.1.5. Warning¶
The combine_echodata step in this notebook may result in kernel dying on Binder or an older computer. We are actively working on improving memory handling especially on distributed systems.
3.1.6. Note¶
We encourage importing echopype as ep for consistency.
from pathlib import Path
import itertools as it
import datetime as dt
from dateutil import parser as dtparser
import fsspec
import xarray as xr
import matplotlib.pyplot as plt
import hvplot.xarray
import echopype as ep
import warnings
warnings.simplefilter("ignore", category=DeprecationWarning)
3.2. Establish connection with the OOI Raw Data Archive and generate list of target EK60 .raw files¶
Access and inspect the publicly accessible OOI Raw Data Archive (an HTTP server) as if it were a local file system. This will be done through the Python fsspec file system and bytes storage interface. We will use fsspec.filesystem.glob (fs.glob) to generate a list of all EK60 .raw data files in the archive then filter on file names for target dates of interest.
fs = fsspec.filesystem('https')
ooi_raw_url = (
"https://rawdata.oceanobservatories.org/files/"
"CE04OSPS/PC01B/ZPLSCB102_10.33.10.143/2017/08"
)
Now let’s specify the range of dates we will be pulling data from. Note that the data filenames contain the time information but were recorded at UTC time.
def in_range(raw_file: str, start: dt.datetime, end: dt.datetime) -> bool:
"""Check if file url is in datetime range"""
file_name = Path(raw_file).name
# file_name = os.path.basename(raw_file)
file_datetime = dtparser.parse(file_name, fuzzy=True)
return file_datetime >= start and file_datetime <= end
start_datetime = dt.datetime(2017, 8, 21, 7, 0)
end_datetime = dt.datetime(2017, 8, 22, 7, 0)
On the OOI Raw Data Archive, the monthly folder is further split to daily folders, so we can simply grab data from the desired days.
desired_day_urls = [f"{ooi_raw_url}/{day}" for day in range(start_datetime.day, end_datetime.day + 1)]
desired_day_urls
['https://rawdata.oceanobservatories.org/files/CE04OSPS/PC01B/ZPLSCB102_10.33.10.143/2017/08/21',
'https://rawdata.oceanobservatories.org/files/CE04OSPS/PC01B/ZPLSCB102_10.33.10.143/2017/08/22']
Grab all raw files within daily folders by using the filesytem glob, just like the Linux glob.
all_raw_file_urls = it.chain.from_iterable([fs.glob(f"{day_url}/*.raw") for day_url in desired_day_urls])
dt.timedelta(hours=3)
datetime.timedelta(seconds=10800)
desired_raw_file_urls = list(filter(
lambda raw_file: in_range(
raw_file,
start_datetime-dt.timedelta(hours=3), # 3 hour buffer to select files
end_datetime+dt.timedelta(hours=3)
),
all_raw_file_urls
))
print(f"There are {len(desired_raw_file_urls)} raw files within the specified datetime range.")
There are 19 raw files within the specified datetime range.
3.3. Process the archived raw files with echopype¶
3.3.1. Examine the workflow by processing just one file¶
Let’s first test the echopype workflow by converting and processing 1 file from the above list.
We will use ep.open_raw to directly read in a raw data file from the OOI HTTP server.
The type of sonar needs to be specified as an input argument. The echosounders on the OOI Regional Cabled Array are Simrad EK60 echosounder. All other uncabled echosounders are the Acoustic Zooplankton and Fisher Profiler (AZFP) manufacturered by ASL Environmental Sciences. Echopype supports both of these and other instruments (see echopype documentation for detail).
3.3.2. Converting from raw data files to a standardized data format¶
Below we already know the path to the 1 file on the http server:
echodata = ep.open_raw(raw_file=desired_raw_file_urls[0], sonar_model="ek60")
20:02:46 parsing file OOI-D20170821-T045717.raw, time of first ping: 2017-Aug-21 04:57:17
Here echopype read, parse, and convert content of the raw file into memory, and gives you a nice representation of the converted file below as a Python EchoData object.
echodata
EchoData: standardized raw data from Internal Memory
-
- conventions :
- CF-1.7, SONAR-netCDF4-1.0, ACDD-1.3
- keywords :
- EK60
- sonar_convention_authority :
- ICES
- sonar_convention_name :
- SONAR-netCDF4
- sonar_convention_version :
- 1.0
- summary :
- title :
- date_created :
- 2017-08-21T04:57:17Z
- survey_name :
<xarray.Dataset> Dimensions: () Data variables: *empty* Attributes: conventions: CF-1.7, SONAR-netCDF4-1.0, ACDD-1.3 keywords: EK60 sonar_convention_authority: ICES sonar_convention_name: SONAR-netCDF4 sonar_convention_version: 1.0 summary: title: date_created: 2017-08-21T04:57:17Z survey_name:xarray.Dataset -
- frequency: 3
- ping_time: 5923
- frequency(frequency)float643.8e+04 1.2e+05 2e+05
- units :
- Hz
- long_name :
- Transducer frequency
- valid_min :
- 0.0
array([ 38000., 120000., 200000.])
- ping_time(ping_time)datetime64[ns]2017-08-21T04:57:17.328999936 .....
- axis :
- T
- long_name :
- Timestamps for NMEA position datagrams
- standard_name :
- time
array(['2017-08-21T04:57:17.328999936', '2017-08-21T04:57:18.332000256', '2017-08-21T04:57:19.335000064', ..., '2017-08-21T06:36:15.549000192', '2017-08-21T06:36:16.552000000', '2017-08-21T06:36:17.555999744'], dtype='datetime64[ns]')
- absorption_indicative(frequency, ping_time)float640.009785 0.009785 ... 0.05269
- long_name :
- Indicative acoustic absorption
- units :
- dB/m
- valid_min :
- 0.0
array([[0.00978527, 0.00978527, 0.00978527, ..., 0.00978527, 0.00978527, 0.00978527], [0.03744031, 0.03744031, 0.03744031, ..., 0.03744031, 0.03744031, 0.03744031], [0.05268759, 0.05268759, 0.05268759, ..., 0.05268759, 0.05268759, 0.05268759]]) - sound_speed_indicative(frequency, ping_time)float641.494e+03 1.494e+03 ... 1.494e+03
- long_name :
- Indicative sound speed
- standard_name :
- speed_of_sound_in_sea_water
- units :
- m/s
- valid_min :
- 0.0
array([[1493.88879395, 1493.88879395, 1493.88879395, ..., 1493.88879395, 1493.88879395, 1493.88879395], [1493.88879395, 1493.88879395, 1493.88879395, ..., 1493.88879395, 1493.88879395, 1493.88879395], [1493.88879395, 1493.88879395, 1493.88879395, ..., 1493.88879395, 1493.88879395, 1493.88879395]])
<xarray.Dataset> Dimensions: (frequency: 3, ping_time: 5923) Coordinates: * frequency (frequency) float64 3.8e+04 1.2e+05 2e+05 * ping_time (ping_time) datetime64[ns] 2017-08-21T04:57:17.32... Data variables: absorption_indicative (frequency, ping_time) float64 0.009785 ... 0.05269 sound_speed_indicative (frequency, ping_time) float64 1.494e+03 ... 1.49...xarray.Dataset -
- location_time: 1
- frequency: 3
- ping_time: 5923
- location_time(location_time)datetime64[ns]NaT
- axis :
- T
- long_name :
- Timestamps for NMEA position datagrams
- standard_name :
- time
array(['NaT'], dtype='datetime64[ns]')
- frequency(frequency)float643.8e+04 1.2e+05 2e+05
- units :
- Hz
- long_name :
- Transducer frequency
- valid_min :
- 0.0
array([ 38000., 120000., 200000.])
- ping_time(ping_time)datetime64[ns]2017-08-21T04:57:17.328999936 .....
- axis :
- T
- long_name :
- Timestamps for position datagrams
- standard_name :
- time
array(['2017-08-21T04:57:17.328999936', '2017-08-21T04:57:18.332000256', '2017-08-21T04:57:19.335000064', ..., '2017-08-21T06:36:15.549000192', '2017-08-21T06:36:16.552000000', '2017-08-21T06:36:17.555999744'], dtype='datetime64[ns]')
- latitude(location_time)float64dask.array<chunksize=(1,), meta=np.ndarray>
- long_name :
- Platform latitude
- standard_name :
- latitude
- units :
- degrees_north
- valid_range :
- (-90.0, 90.0)
Array Chunk Bytes 8 B 8 B Shape (1,) (1,) Count 2 Tasks 1 Chunks Type float64 numpy.ndarray - longitude(location_time)float64dask.array<chunksize=(1,), meta=np.ndarray>
- long_name :
- Platform longitude
- standard_name :
- longitude
- units :
- degrees_east
- valid_range :
- (-180.0, 180.0)
Array Chunk Bytes 8 B 8 B Shape (1,) (1,) Count 2 Tasks 1 Chunks Type float64 numpy.ndarray - sentence_type(location_time)float64dask.array<chunksize=(1,), meta=np.ndarray>
Array Chunk Bytes 8 B 8 B Shape (1,) (1,) Count 2 Tasks 1 Chunks Type float64 numpy.ndarray - pitch(frequency, ping_time)float64dask.array<chunksize=(3, 2500), meta=np.ndarray>
- long_name :
- Platform pitch
- standard_name :
- platform_pitch_angle
- units :
- arc_degree
- valid_range :
- (-90.0, 90.0)
Array Chunk Bytes 138.82 kiB 58.59 kiB Shape (3, 5923) (3, 2500) Count 3 Tasks 3 Chunks Type float64 numpy.ndarray - roll(frequency, ping_time)float64dask.array<chunksize=(3, 2500), meta=np.ndarray>
- long_name :
- Platform roll
- standard_name :
- platform_roll_angle
- units :
- arc_degree
- valid_range :
- (-90.0, 90.0)
Array Chunk Bytes 138.82 kiB 58.59 kiB Shape (3, 5923) (3, 2500) Count 3 Tasks 3 Chunks Type float64 numpy.ndarray - heave(frequency, ping_time)float64dask.array<chunksize=(3, 2500), meta=np.ndarray>
- long_name :
- Platform heave
- standard_name :
- platform_heave_angle
- units :
- arc_degree
- valid_range :
- (-90.0, 90.0)
Array Chunk Bytes 138.82 kiB 58.59 kiB Shape (3, 5923) (3, 2500) Count 3 Tasks 3 Chunks Type float64 numpy.ndarray - water_level(frequency, ping_time)float64dask.array<chunksize=(3, 2500), meta=np.ndarray>
- long_name :
- z-axis distance from the platform coordinate system origin to the sonar transducer
- units :
- m
Array Chunk Bytes 138.82 kiB 58.59 kiB Shape (3, 5923) (3, 2500) Count 3 Tasks 3 Chunks Type float64 numpy.ndarray
<xarray.Dataset> Dimensions: (location_time: 1, frequency: 3, ping_time: 5923) Coordinates: * location_time (location_time) datetime64[ns] NaT * frequency (frequency) float64 3.8e+04 1.2e+05 2e+05 * ping_time (ping_time) datetime64[ns] 2017-08-21T04:57:17.328999936 .... Data variables: latitude (location_time) float64 dask.array<chunksize=(1,), meta=np.ndarray> longitude (location_time) float64 dask.array<chunksize=(1,), meta=np.ndarray> sentence_type (location_time) float64 dask.array<chunksize=(1,), meta=np.ndarray> pitch (frequency, ping_time) float64 dask.array<chunksize=(3, 2500), meta=np.ndarray> roll (frequency, ping_time) float64 dask.array<chunksize=(3, 2500), meta=np.ndarray> heave (frequency, ping_time) float64 dask.array<chunksize=(3, 2500), meta=np.ndarray> water_level (frequency, ping_time) float64 dask.array<chunksize=(3, 2500), meta=np.ndarray>xarray.Dataset -
- location_time: 1
- location_time(location_time)datetime64[ns]2017-08-21T04:57:17.328999936
- axis :
- T
- long_name :
- Timestamps for NMEA datagrams
- standard_name :
- time
array(['2017-08-21T04:57:17.328999936'], dtype='datetime64[ns]')
- NMEA_datagram(location_time)<U22'$SDVLW,0.000,N,0.000,N'
- long_name :
- NMEA datagram
array(['$SDVLW,0.000,N,0.000,N'], dtype='<U22')
- description :
- All NMEA sensor datagrams
<xarray.Dataset> Dimensions: (location_time: 1) Coordinates: * location_time (location_time) datetime64[ns] 2017-08-21T04:57:17.328999936 Data variables: NMEA_datagram (location_time) <U22 '$SDVLW,0.000,N,0.000,N' Attributes: description: All NMEA sensor datagramsxarray.Dataset -
- conversion_software_name :
- echopype
- conversion_software_version :
- 0.5.4
- conversion_time :
- 2021-10-28T03:02:50Z
- src_filenames :
- https://rawdata.oceanobservatories.org/files/CE04OSPS/PC01B/ZPLSCB102_10.33.10.143/2017/08/21/OOI-D20170821-T045717.raw
- duplicate_ping_times :
- 0
<xarray.Dataset> Dimensions: () Data variables: *empty* Attributes: conversion_software_name: echopype conversion_software_version: 0.5.4 conversion_time: 2021-10-28T03:02:50Z src_filenames: https://rawdata.oceanobservatories.org/file... duplicate_ping_times: 0xarray.Dataset -
- sonar_manufacturer :
- Simrad
- sonar_model :
- ER60
- sonar_serial_number :
- sonar_software_name :
- sonar_software_version :
- 2.4.3
- sonar_type :
- echosounder
<xarray.Dataset> Dimensions: () Data variables: *empty* Attributes: sonar_manufacturer: Simrad sonar_model: ER60 sonar_serial_number: sonar_software_name: sonar_software_version: 2.4.3 sonar_type: echosounderxarray.Dataset -
- frequency: 3
- ping_time: 5923
- range_bin: 1072
- frequency(frequency)float643.8e+04 1.2e+05 2e+05
- units :
- Hz
- long_name :
- Transducer frequency
- valid_min :
- 0.0
array([ 38000., 120000., 200000.])
- ping_time(ping_time)datetime64[ns]2017-08-21T04:57:17.328999936 .....
- axis :
- T
- long_name :
- Timestamp of each ping
- standard_name :
- time
array(['2017-08-21T04:57:17.328999936', '2017-08-21T04:57:18.332000256', '2017-08-21T04:57:19.335000064', ..., '2017-08-21T06:36:15.549000192', '2017-08-21T06:36:16.552000000', '2017-08-21T06:36:17.555999744'], dtype='datetime64[ns]') - range_bin(range_bin)int640 1 2 3 4 ... 1068 1069 1070 1071
array([ 0, 1, 2, ..., 1069, 1070, 1071])
- channel_id(frequency)<U39'GPT 38 kHz 00907208dd13 5-1 OO...
array(['GPT 38 kHz 00907208dd13 5-1 OOI.38|200', 'GPT 120 kHz 00907208a0b1 3-1 ES120-7CD', 'GPT 200 kHz 00907208dd13 5-2 OOI38|200'], dtype='<U39') - beam_type(frequency)int640 1 0
- long_name :
- type of transducer (0-single, 1-split)
array([0, 1, 0])
- beamwidth_receive_alongship(frequency)float647.1 7.0 7.0
- long_name :
- Half power one-way receive beam width along alongship axis of beam
- units :
- arc_degree
- valid_range :
- (0.0, 360.0)
array([7.0999999, 7. , 7. ])
- beamwidth_receive_athwartship(frequency)float647.1 7.0 7.0
- long_name :
- Half power one-way receive beam width along athwartship axis of beam
- units :
- arc_degree
- valid_range :
- (0.0, 360.0)
array([7.0999999, 7. , 7. ])
- beamwidth_transmit_alongship(frequency)float647.1 7.0 7.0
- long_name :
- Half power one-way transmit beam width along alongship axis of beam
- units :
- arc_degree
- valid_range :
- (0.0, 360.0)
array([7.0999999, 7. , 7. ])
- beamwidth_transmit_athwartship(frequency)float647.1 7.0 7.0
- long_name :
- Half power one-way transmit beam width along athwartship axis of beam
- units :
- arc_degree
- valid_range :
- (0.0, 360.0)
array([7.0999999, 7. , 7. ])
- beam_direction_x(frequency)float640.0 0.0 0.0
- long_name :
- x-component of the vector that gives the pointing direction of the beam, in sonar beam coordinate system
- units :
- 1
- valid_range :
- (-1.0, 1.0)
array([0., 0., 0.])
- beam_direction_y(frequency)float640.0 0.0 0.0
- long_name :
- y-component of the vector that gives the pointing direction of the beam, in sonar beam coordinate system
- units :
- 1
- valid_range :
- (-1.0, 1.0)
array([0., 0., 0.])
- beam_direction_z(frequency)float640.0 0.0 0.0
- long_name :
- z-component of the vector that gives the pointing direction of the beam, in sonar beam coordinate system
- units :
- 1
- valid_range :
- (-1.0, 1.0)
array([0., 0., 0.])
- angle_offset_alongship(frequency)float640.0 0.0 0.0
- long_name :
- electrical alongship angle of the transducer
array([0., 0., 0.])
- angle_offset_athwartship(frequency)float640.0 0.0 0.0
- long_name :
- electrical athwartship angle of the transducer
array([0., 0., 0.])
- angle_sensitivity_alongship(frequency)float6421.9 23.0 23.0
- long_name :
- alongship sensitivity of the transducer
array([21.89999962, 23. , 23. ])
- angle_sensitivity_athwartship(frequency)float6421.9 23.0 23.0
- long_name :
- athwartship sensitivity of the transducer
array([21.89999962, 23. , 23. ])
- equivalent_beam_angle(frequency)float64-20.6 -20.7 -20.7
- long_name :
- Equivalent beam angle
- units :
- sr
- valid_range :
- (0.0, 12.566370614359172)
array([-20.60000038, -20.70000076, -20.70000076])
- transducer_offset_x(frequency)float640.0 0.0 0.0
- long_name :
- x-axis distance from the platform coordinate system origin to the sonar transducer
- units :
- m
array([0., 0., 0.])
- transducer_offset_y(frequency)float640.0 0.0 0.0
- long_name :
- y-axis distance from the platform coordinate system origin to the sonar transducer
- units :
- m
array([0., 0., 0.])
- transducer_offset_z(frequency)float640.0 0.0 0.0
- long_name :
- z-axis distance from the platform coordinate system origin to the sonar transducer
- units :
- m
array([0., 0., 0.])
- gain_correction(frequency)float6426.5 25.0 25.0
- long_name :
- Gain correction
- units :
- dB
array([26.5, 25. , 25. ])
- gpt_software_version(frequency)<U6'150120' '070413' '150120'
array(['150120', '070413', '150120'], dtype='<U6')
- backscatter_r(frequency, ping_time, range_bin)float3211.97 19.57 19.36 ... -134.3 -135.7
- long_name :
- Backscatter power
- units :
- dB
array([[[ 11.970646, 19.56695 , 19.355288, ..., -98.787224, -98.164 , -99.70443 ], [ 11.970646, 19.56695 , 19.355288, ..., -105.54864 , -103.78479 , -101.65642 ], [ 11.958887, 19.56695 , 19.343529, ..., -102.608894, -101.691696, -99.63387 ], ..., [ 11.982405, 19.578709, 19.355288, ..., -102.72649 , -115.238045, -119.153786], [ 11.982405, 19.578709, 19.355288, ..., -101.75049 , -101.46828 , -107.31249 ], [ 11.958887, 19.555191, 19.343529, ..., -103.667206, -109.51142 , -119.61239 ]], [[ 12.076477, 17.908934, 17.885414, ..., -123.75155 , -116.00238 , -110.32279 ], [ 12.076477, 17.908934, 17.885414, ..., -112.06312 , -119.31841 , -120.635414], [ 12.088236, 17.908934, 17.885414, ..., -114.03863 , -117.43697 , -118.530556], ... [ 12.076477, 17.908934, 17.873655, ..., -114.66185 , -114.47371 , -116.8843 ], [ 12.076477, 17.908934, 17.885414, ..., -111.59276 , -121.2704 , -117.58984 ], [ 12.076477, 17.908934, 17.873655, ..., -114.26205 , -117.07245 , -118.0602 ]], [[-124.17487 , -141.24892 , -141.15485 , ..., -149.09216 , -147.79868 , -135.38118 ], [-133.01762 , -141.20187 , -141.15485 , ..., -130.92453 , -127.243965, -127.23221 ], [-127.66729 , -141.1078 , -141.17836 , ..., -133.47623 , -132.74718 , -132.10043 ], ..., [-136.98041 , -141.15485 , -141.17836 , ..., -147.15193 , -132.57079 , -131.15971 ], [-124.82162 , -141.03726 , -141.17836 , ..., -134.88731 , -133.89955 , -133.14697 ], [-131.7947 , -141.17836 , -141.17836 , ..., -132.02988 , -134.25232 , -135.7222 ]]], dtype=float32) - sample_interval(frequency, ping_time)float640.000256 0.000256 ... 0.000256
- long_name :
- Interval between recorded raw data samples
- units :
- s
- valid_min :
- 0.0
array([[0.000256, 0.000256, 0.000256, ..., 0.000256, 0.000256, 0.000256], [0.000256, 0.000256, 0.000256, ..., 0.000256, 0.000256, 0.000256], [0.000256, 0.000256, 0.000256, ..., 0.000256, 0.000256, 0.000256]]) - transmit_bandwidth(frequency, ping_time)float642.425e+03 2.425e+03 ... 3.088e+03
- long_name :
- Nominal bandwidth of transmitted pulse
- units :
- Hz
- valid_min :
- 0.0
array([[2425.1496582 , 2425.1496582 , 2425.1496582 , ..., 2425.1496582 , 2425.1496582 , 2425.1496582 ], [3026.39160156, 3026.39160156, 3026.39160156, ..., 3026.39160156, 3026.39160156, 3026.39160156], [3088.40039062, 3088.40039062, 3088.40039062, ..., 3088.40039062, 3088.40039062, 3088.40039062]]) - transmit_duration_nominal(frequency, ping_time)float640.001024 0.001024 ... 0.001024
- long_name :
- Nominal bandwidth of transmitted pulse
- units :
- s
- valid_min :
- 0.0
array([[0.001024, 0.001024, 0.001024, ..., 0.001024, 0.001024, 0.001024], [0.001024, 0.001024, 0.001024, ..., 0.001024, 0.001024, 0.001024], [0.001024, 0.001024, 0.001024, ..., 0.001024, 0.001024, 0.001024]]) - transmit_power(frequency, ping_time)float64500.0 500.0 500.0 ... 150.0 150.0
- long_name :
- Nominal transmit power
- units :
- W
- valid_min :
- 0.0
array([[500., 500., 500., ..., 500., 500., 500.], [250., 250., 250., ..., 250., 250., 250.], [150., 150., 150., ..., 150., 150., 150.]]) - data_type(frequency, ping_time)float641.0 1.0 1.0 1.0 ... 1.0 1.0 1.0 1.0
- long_name :
- recorded data type (1-power only, 2-angle only 3-power and angle)
array([[1., 1., 1., ..., 1., 1., 1.], [3., 3., 3., ..., 3., 3., 3.], [1., 1., 1., ..., 1., 1., 1.]]) - count(frequency, ping_time)float641.072e+03 1.072e+03 ... 1.072e+03
- long_name :
- Number of samples
array([[1072., 1072., 1072., ..., 1072., 1072., 1072.], [1072., 1072., 1072., ..., 1072., 1072., 1072.], [1072., 1072., 1072., ..., 1072., 1072., 1072.]]) - offset(frequency, ping_time)float640.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0
- long_name :
- Offset of first sample
array([[0., 0., 0., ..., 0., 0., 0.], [0., 0., 0., ..., 0., 0., 0.], [0., 0., 0., ..., 0., 0., 0.]]) - transmit_mode(frequency, ping_time)float640.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0
- long_name :
- 0 = Active, 1 = Passive, 2 = Test, -1 = Unknown
array([[0., 0., 0., ..., 0., 0., 0.], [0., 0., 0., ..., 0., 0., 0.], [0., 0., 0., ..., 0., 0., 0.]]) - angle_athwartship(frequency, ping_time, range_bin)float64nan nan nan nan ... nan nan nan nan
- long_name :
- electrical athwartship angle
array([[[ nan, nan, nan, ..., nan, nan, nan], [ nan, nan, nan, ..., nan, nan, nan], [ nan, nan, nan, ..., nan, nan, nan], ..., [ nan, nan, nan, ..., nan, nan, nan], [ nan, nan, nan, ..., nan, nan, nan], [ nan, nan, nan, ..., nan, nan, nan]], [[ -1., -2., -2., ..., -118., -128., -85.], [ -2., -2., -2., ..., 18., 103., -123.], [ -1., -2., -2., ..., -111., -108., -80.], ..., [ -1., -2., -2., ..., -112., 4., 104.], [ -2., -2., -2., ..., -37., -58., -54.], [ -1., -2., -2., ..., 24., -29., -100.]], [[ nan, nan, nan, ..., nan, nan, nan], [ nan, nan, nan, ..., nan, nan, nan], [ nan, nan, nan, ..., nan, nan, nan], ..., [ nan, nan, nan, ..., nan, nan, nan], [ nan, nan, nan, ..., nan, nan, nan], [ nan, nan, nan, ..., nan, nan, nan]]]) - angle_alongship(frequency, ping_time, range_bin)float64nan nan nan nan ... nan nan nan nan
- long_name :
- electrical alongship angle
array([[[ nan, nan, nan, ..., nan, nan, nan], [ nan, nan, nan, ..., nan, nan, nan], [ nan, nan, nan, ..., nan, nan, nan], ..., [ nan, nan, nan, ..., nan, nan, nan], [ nan, nan, nan, ..., nan, nan, nan], [ nan, nan, nan, ..., nan, nan, nan]], [[ -1., -2., -2., ..., 107., 92., 58.], [ -1., -2., -2., ..., -22., -40., -10.], [ -1., -2., -2., ..., -29., -37., 40.], ..., [ -1., -2., -2., ..., -30., -57., -96.], [ -1., -2., -2., ..., 20., -55., 111.], [ -1., -2., -2., ..., -80., -93., 73.]], [[ nan, nan, nan, ..., nan, nan, nan], [ nan, nan, nan, ..., nan, nan, nan], [ nan, nan, nan, ..., nan, nan, nan], ..., [ nan, nan, nan, ..., nan, nan, nan], [ nan, nan, nan, ..., nan, nan, nan], [ nan, nan, nan, ..., nan, nan, nan]]])
- beam_mode :
- vertical
- conversion_equation_t :
- type_3
<xarray.Dataset> Dimensions: (frequency: 3, ping_time: 5923, range_bin: 1072) Coordinates: * frequency (frequency) float64 3.8e+04 1.2e+05 2e+05 * ping_time (ping_time) datetime64[ns] 2017-08-21T04:... * range_bin (range_bin) int64 0 1 2 3 ... 1069 1070 1071 Data variables: (12/30) channel_id (frequency) <U39 'GPT 38 kHz 00907208dd1... beam_type (frequency) int64 0 1 0 beamwidth_receive_alongship (frequency) float64 7.1 7.0 7.0 beamwidth_receive_athwartship (frequency) float64 7.1 7.0 7.0 beamwidth_transmit_alongship (frequency) float64 7.1 7.0 7.0 beamwidth_transmit_athwartship (frequency) float64 7.1 7.0 7.0 ... ... data_type (frequency, ping_time) float64 1.0 ... 1.0 count (frequency, ping_time) float64 1.072e+03 ... offset (frequency, ping_time) float64 0.0 ... 0.0 transmit_mode (frequency, ping_time) float64 0.0 ... 0.0 angle_athwartship (frequency, ping_time, range_bin) float64 ... angle_alongship (frequency, ping_time, range_bin) float64 ... Attributes: beam_mode: vertical conversion_equation_t: type_3xarray.Dataset -
- frequency: 3
- pulse_length_bin: 5
- frequency(frequency)float641.2e+05 3.8e+04 2e+05
- units :
- Hz
- long_name :
- Transducer frequency
- valid_min :
- 0.0
array([120000., 38000., 200000.])
- pulse_length_bin(pulse_length_bin)int640 1 2 3 4
array([0, 1, 2, 3, 4])
- sa_correction(frequency, pulse_length_bin)float640.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0
array([[0., 0., 0., 0., 0.], [0., 0., 0., 0., 0.], [0., 0., 0., 0., 0.]]) - gain_correction(frequency, pulse_length_bin)float6423.5 24.8 25.0 ... 25.0 25.0 25.0
array([[23.5 , 24.799999, 25. , 25. , 25. ], [24. , 26. , 26.5 , 26.5 , 26.5 ], [23.5 , 24.799999, 25. , 25. , 25. ]]) - pulse_length(frequency, pulse_length_bin)float646.4e-05 0.000128 ... 0.001024
array([[6.400e-05, 1.280e-04, 2.560e-04, 5.120e-04, 1.024e-03], [2.560e-04, 5.120e-04, 1.024e-03, 2.048e-03, 4.096e-03], [6.400e-05, 1.280e-04, 2.560e-04, 5.120e-04, 1.024e-03]])
<xarray.Dataset> Dimensions: (frequency: 3, pulse_length_bin: 5) Coordinates: * frequency (frequency) float64 1.2e+05 3.8e+04 2e+05 * pulse_length_bin (pulse_length_bin) int64 0 1 2 3 4 Data variables: sa_correction (frequency, pulse_length_bin) float64 0.0 0.0 ... 0.0 0.0 gain_correction (frequency, pulse_length_bin) float64 23.5 24.8 ... 25.0 pulse_length (frequency, pulse_length_bin) float64 6.4e-05 ... 0.001024xarray.Dataset
The EchoData object can be saved to either the netCDF4 or zarr formats through to_netcdf or to_zarr methods.
# Create directories for files genereated in this notebook.
base_dpath = Path('./exports')
base_dpath.mkdir(exist_ok=True)
output_dpath = Path(base_dpath / 'ooimooring_onefiletest')
output_dpath.mkdir(exist_ok=True)
# Save to netCDF format
echodata.to_netcdf(save_path=output_dpath, overwrite=True)
20:02:52 overwriting exports/ooimooring_onefiletest/OOI-D20170821-T045717.nc
# Save to zarrr format
echodata.to_zarr(save_path=output_dpath, overwrite=True)
20:03:00 overwriting exports/ooimooring_onefiletest/OOI-D20170821-T045717.zarr
3.3.3. Basic echo processing¶
At present echopype supports basic processing funcionalities including calibration (from raw instrument data records to volume backscattering strength, \(S_V\)), denoising, and computing mean volume backscattering strength, \(\overline{S_V}\) or \(\text{MVBS}\). The Echodata object can be passed into various calibrate and preprocessing functions without having to write out any intermediate files.
Here we demonstrate calibration to obtain \(S_V\). For EK60 data, by default the function uses environmental (sound speed and absorption) and calibration parameters stored in the data file. Users can optionally specify other parameter choices.
# Compute volume backscattering strength (Sv) from raw data
ds_Sv = ep.calibrate.compute_Sv(echodata)
The computed Sv is stored with other variables used in the calibration operation.
ds_Sv
<xarray.Dataset>
Dimensions: (frequency: 3, ping_time: 5923, range_bin: 1072)
Coordinates:
* frequency (frequency) float64 3.8e+04 1.2e+05 2e+05
* ping_time (ping_time) datetime64[ns] 2017-08-21T04:57:17.328...
* range_bin (range_bin) int64 0 1 2 3 4 ... 1068 1069 1070 1071
Data variables:
Sv (frequency, ping_time, range_bin) float64 3.839 .....
range (frequency, ping_time, range_bin) float64 0.0 ... ...
temperature object None
salinity object None
pressure object None
sound_speed (frequency, ping_time) float64 1.494e+03 ... 1.494...
sound_absorption (frequency, ping_time) float64 0.009785 ... 0.05269
sa_correction (frequency) float64 0.0 0.0 0.0
gain_correction (frequency) float64 26.5 25.0 25.0
equivalent_beam_angle (frequency) float64 -20.6 -20.7 -20.7xarray.Dataset
- frequency: 3
- ping_time: 5923
- range_bin: 1072
- frequency(frequency)float643.8e+04 1.2e+05 2e+05
- units :
- Hz
- long_name :
- Transducer frequency
- valid_min :
- 0.0
array([ 38000., 120000., 200000.])
- ping_time(ping_time)datetime64[ns]2017-08-21T04:57:17.328999936 .....
- axis :
- T
- long_name :
- Timestamps for NMEA position datagrams
- standard_name :
- time
array(['2017-08-21T04:57:17.328999936', '2017-08-21T04:57:18.332000256', '2017-08-21T04:57:19.335000064', ..., '2017-08-21T06:36:15.549000192', '2017-08-21T06:36:16.552000000', '2017-08-21T06:36:17.555999744'], dtype='datetime64[ns]') - range_bin(range_bin)int640 1 2 3 4 ... 1068 1069 1070 1071
array([ 0, 1, 2, ..., 1069, 1070, 1071])
- Sv(frequency, ping_time, range_bin)float643.839 11.43 11.22 ... -51.91 -53.35
array([[[ 3.83856777, 11.43487171, 11.22320942, ..., -56.73248439, -56.09738314, -57.62594001], [ 3.83856777, 11.43487171, 11.22320942, ..., -63.49389796, -61.71817171, -59.57792823], [ 3.82680897, 11.43487171, 11.21145061, ..., -60.55415492, -59.62507784, -57.55538337], ..., [ 3.85032658, 11.44663052, 11.22320942, ..., -60.67174678, -73.17142641, -77.07529609], [ 3.85032658, 11.44663052, 11.22320942, ..., -59.69574885, -59.40165865, -65.23400276], [ 3.82680897, 11.42311291, 11.21145061, ..., -61.61246638, -67.44480288, -77.533899 ]], [[ 20.04265224, 25.87510883, 25.85158932, ..., -54.31367532, -46.54205175, -40.84001588], [ 20.04265224, 25.87510883, 25.85158932, ..., -42.62524453, -49.85808416, -51.15263795], [ 20.05441105, 25.87510883, 25.85158932, ..., -44.60075418, -47.97664495, -49.04777955], ... [ 20.04265224, 25.87510883, 25.83983051, ..., -45.22397653, -45.01338049, -47.40152406], [ 20.04265224, 25.87510883, 25.85158932, ..., -42.15488473, -51.81007238, -48.10706757], [ 20.04265224, 25.87510883, 25.83983051, ..., -44.82417336, -47.61212011, -48.57742738]], [[-109.55323567, -126.62727894, -126.53320851, ..., -66.77704258, -65.45526876, -53.00949437], [-118.39598622, -126.5802361 , -126.53320851, ..., -48.60940952, -44.90055837, -44.86052282], [-113.04565205, -126.48616566, -126.5567223 , ..., -51.1611063 , -50.40377035, -49.72874791], ..., [-122.35877003, -126.53320851, -126.5567223 , ..., -64.83681126, -50.22737874, -48.78802831], [-110.19997944, -126.41562428, -126.5567223 , ..., -52.57219334, -51.55614461, -50.77528722], [-117.17305531, -126.5567223 , -126.5567223 , ..., -49.7147562 , -51.90891256, -53.35051305]]]) - range(frequency, ping_time, range_bin)float640.0 0.0 0.0 ... 204.0 204.2 204.4
array([[[ 0. , 0. , 0. , ..., 204.02935541, 204.22057317, 204.41179094], [ 0. , 0. , 0. , ..., 204.02935541, 204.22057317, 204.41179094], [ 0. , 0. , 0. , ..., 204.02935541, 204.22057317, 204.41179094], ..., [ 0. , 0. , 0. , ..., 204.02935541, 204.22057317, 204.41179094], [ 0. , 0. , 0. , ..., 204.02935541, 204.22057317, 204.41179094], [ 0. , 0. , 0. , ..., 204.02935541, 204.22057317, 204.41179094]], [[ 0. , 0. , 0. , ..., 204.02935541, 204.22057317, 204.41179094], [ 0. , 0. , 0. , ..., 204.02935541, 204.22057317, 204.41179094], [ 0. , 0. , 0. , ..., 204.02935541, 204.22057317, 204.41179094], ... [ 0. , 0. , 0. , ..., 204.02935541, 204.22057317, 204.41179094], [ 0. , 0. , 0. , ..., 204.02935541, 204.22057317, 204.41179094], [ 0. , 0. , 0. , ..., 204.02935541, 204.22057317, 204.41179094]], [[ 0. , 0. , 0. , ..., 204.02935541, 204.22057317, 204.41179094], [ 0. , 0. , 0. , ..., 204.02935541, 204.22057317, 204.41179094], [ 0. , 0. , 0. , ..., 204.02935541, 204.22057317, 204.41179094], ..., [ 0. , 0. , 0. , ..., 204.02935541, 204.22057317, 204.41179094], [ 0. , 0. , 0. , ..., 204.02935541, 204.22057317, 204.41179094], [ 0. , 0. , 0. , ..., 204.02935541, 204.22057317, 204.41179094]]]) - temperature()objectNone
array(None, dtype=object)
- salinity()objectNone
array(None, dtype=object)
- pressure()objectNone
array(None, dtype=object)
- sound_speed(frequency, ping_time)float641.494e+03 1.494e+03 ... 1.494e+03
- long_name :
- Indicative sound speed
- standard_name :
- speed_of_sound_in_sea_water
- units :
- m/s
- valid_min :
- 0.0
array([[1493.88879395, 1493.88879395, 1493.88879395, ..., 1493.88879395, 1493.88879395, 1493.88879395], [1493.88879395, 1493.88879395, 1493.88879395, ..., 1493.88879395, 1493.88879395, 1493.88879395], [1493.88879395, 1493.88879395, 1493.88879395, ..., 1493.88879395, 1493.88879395, 1493.88879395]]) - sound_absorption(frequency, ping_time)float640.009785 0.009785 ... 0.05269
- long_name :
- Indicative acoustic absorption
- units :
- dB/m
- valid_min :
- 0.0
array([[0.00978527, 0.00978527, 0.00978527, ..., 0.00978527, 0.00978527, 0.00978527], [0.03744031, 0.03744031, 0.03744031, ..., 0.03744031, 0.03744031, 0.03744031], [0.05268759, 0.05268759, 0.05268759, ..., 0.05268759, 0.05268759, 0.05268759]]) - sa_correction(frequency)float640.0 0.0 0.0
array([0., 0., 0.])
- gain_correction(frequency)float6426.5 25.0 25.0
array([26.5, 25. , 25. ])
- equivalent_beam_angle(frequency)float64-20.6 -20.7 -20.7
- long_name :
- Equivalent beam angle
- units :
- sr
- valid_range :
- (0.0, 12.566370614359172)
array([-20.60000038, -20.70000076, -20.70000076])
3.3.4. Quickly visualize the result¶
The default xarray visualization functions are useful in getting a quick sense of the data.
ds_Sv.Sv.sel(frequency=200000).plot.pcolormesh(
x='ping_time', cmap = 'jet', vmin=-80, vmax=-30)
<matplotlib.collections.QuadMesh at 0x161440f40>
Note that the vertical axis is range_bin. This is the bin (or sample) number as recorded in the data. A separate data variable in ds_Sv contains the physical range from the transducer in meters. range has the same dimension as Sv and may not be uniform across all frequency channels or pings, depending on the echosounder setting during data collection.
3.3.5. Convert multiple files and combine into a single EchoData object¶
Now that we verified that echopype does work for a single file, let’s proceed to process all sonar data from August 20-21, 2017.
First, convert all desired files from the OOI HTTP server to a local directory ./exports/ooimooring_allfiles.
# Create a directory for all files
output_dpath = Path(base_dpath / 'ooimooring_allfiles')
output_dpath.mkdir(exist_ok=True)
%%time
for raw_file_url in desired_raw_file_urls:
# Read and convert, resulting in echodata object
ed = ep.open_raw(raw_file=raw_file_url, sonar_model='ek60')
ed.to_zarr(save_path=output_dpath, overwrite=True)
20:03:07 parsing file OOI-D20170821-T045717.raw, time of first ping: 2017-Aug-21 04:57:17
20:03:13 overwriting exports/ooimooring_allfiles/OOI-D20170821-T045717.zarr
20:03:15 parsing file OOI-D20170821-T063618.raw, time of first ping: 2017-Aug-21 06:36:18
20:03:21 overwriting exports/ooimooring_allfiles/OOI-D20170821-T063618.zarr
20:03:24 parsing file OOI-D20170821-T081522.raw, time of first ping: 2017-Aug-21 08:15:22
20:03:35 overwriting exports/ooimooring_allfiles/OOI-D20170821-T081522.zarr
20:03:37 parsing file OOI-D20170821-T095435.raw, time of first ping: 2017-Aug-21 09:54:35
20:03:46 overwriting exports/ooimooring_allfiles/OOI-D20170821-T095435.zarr
20:03:48 parsing file OOI-D20170821-T113343.raw, time of first ping: 2017-Aug-21 11:33:43
20:03:54 overwriting exports/ooimooring_allfiles/OOI-D20170821-T113343.zarr
20:03:56 parsing file OOI-D20170821-T131245.raw, time of first ping: 2017-Aug-21 13:12:45
20:04:03 overwriting exports/ooimooring_allfiles/OOI-D20170821-T131245.zarr
20:04:05 parsing file OOI-D20170821-T145147.raw, time of first ping: 2017-Aug-21 14:51:47
20:04:11 overwriting exports/ooimooring_allfiles/OOI-D20170821-T145147.zarr
20:04:13 parsing file OOI-D20170821-T163049.raw, time of first ping: 2017-Aug-21 16:30:49
20:04:19 overwriting exports/ooimooring_allfiles/OOI-D20170821-T163049.zarr
20:04:22 parsing file OOI-D20170821-T180952.raw, time of first ping: 2017-Aug-21 18:09:52
20:04:30 overwriting exports/ooimooring_allfiles/OOI-D20170821-T180952.zarr
20:04:32 parsing file OOI-D20170821-T194853.raw, time of first ping: 2017-Aug-21 19:48:53
20:04:38 overwriting exports/ooimooring_allfiles/OOI-D20170821-T194853.zarr
20:04:40 parsing file OOI-D20170821-T212802.raw, time of first ping: 2017-Aug-21 21:28:02
20:04:46 overwriting exports/ooimooring_allfiles/OOI-D20170821-T212802.zarr
20:04:48 parsing file OOI-D20170821-T230706.raw, time of first ping: 2017-Aug-21 23:07:06
20:04:51 overwriting exports/ooimooring_allfiles/OOI-D20170821-T230706.zarr
20:04:54 parsing file OOI-D20170822-T000000.raw, time of first ping: 2017-Aug-22 00:00:00
20:05:02 overwriting exports/ooimooring_allfiles/OOI-D20170822-T000000.zarr
20:05:06 parsing file OOI-D20170822-T013902.raw, time of first ping: 2017-Aug-22 01:39:02
20:05:14 overwriting exports/ooimooring_allfiles/OOI-D20170822-T013902.zarr
20:05:17 parsing file OOI-D20170822-T031804.raw, time of first ping: 2017-Aug-22 03:18:04
20:05:24 overwriting exports/ooimooring_allfiles/OOI-D20170822-T031804.zarr
20:05:27 parsing file OOI-D20170822-T045705.raw, time of first ping: 2017-Aug-22 04:57:05
20:05:32 overwriting exports/ooimooring_allfiles/OOI-D20170822-T045705.zarr
20:05:34 parsing file OOI-D20170822-T063606.raw, time of first ping: 2017-Aug-22 06:36:06
20:05:40 overwriting exports/ooimooring_allfiles/OOI-D20170822-T063606.zarr
20:05:42 parsing file OOI-D20170822-T081508.raw, time of first ping: 2017-Aug-22 08:15:08
20:05:48 overwriting exports/ooimooring_allfiles/OOI-D20170822-T081508.zarr
20:05:51 parsing file OOI-D20170822-T095414.raw, time of first ping: 2017-Aug-22 09:54:14
20:05:57 overwriting exports/ooimooring_allfiles/OOI-D20170822-T095414.zarr
CPU times: user 1min 18s, sys: 24.4 s, total: 1min 43s
Wall time: 2min 53s
Then, assemble a list of EchoData object from the converted files. Note that be default the files are lazy-loaded and only metadata are read into memory, until more operations are executed.
# Use fsspec locally to assemble a list of converted files
fs_local = fsspec.filesystem('file')
ed_list = []
for converted_file in fs_local.glob(output_dpath / f"*.zarr"):
ed_list.append(ep.open_converted(converted_file))
Combine all the opened files to a single EchoData object in memory. This will take a bit of time to execute.
ed = ep.combine_echodata(ed_list)
3.3.6. Calibrate the combined EchoData and visualize the mean Sv¶
The single EchoData object is convenient to use for content inspection and calibration.
ds_Sv = ep.calibrate.compute_Sv(ed)
Next, compute the mean Sv (MVBS) with coherent dimensions along physically meaning range (in meters) and ping_time from the calibrated data. This processed dataset is easy to visualize. The average bin size along ping_time can be specified using the offset alias.
ds_MVBS = ep.preprocess.compute_MVBS(
ds_Sv,
range_meter_bin=0.2, # 0.2 meters
ping_time_bin='10S' # 10 seconds
)
The resulting MVBS Dataset has a coherent range coorindate across all frequencies.
ds_MVBS
<xarray.Dataset>
Dimensions: (ping_time: 11017, frequency: 3, range: 1023)
Coordinates:
* ping_time (ping_time) datetime64[ns] 2017-08-21T04:57:10 ... 2017-08-22T...
* frequency (frequency) float64 3.8e+04 1.2e+05 2e+05
* range (range) float64 0.0 0.2 0.4 0.6 0.8 ... 203.8 204.0 204.2 204.4
Data variables:
Sv (frequency, ping_time, range) float64 10.29 2.976 ... -53.92
Attributes:
binning_mode: physical units
range_meter_interval: 0.2m
ping_time_interval: 10Sxarray.Dataset
- ping_time: 11017
- frequency: 3
- range: 1023
- ping_time(ping_time)datetime64[ns]2017-08-21T04:57:10 ... 2017-08-...
array(['2017-08-21T04:57:10.000000000', '2017-08-21T04:57:20.000000000', '2017-08-21T04:57:30.000000000', ..., '2017-08-22T11:32:50.000000000', '2017-08-22T11:33:00.000000000', '2017-08-22T11:33:10.000000000'], dtype='datetime64[ns]') - frequency(frequency)float643.8e+04 1.2e+05 2e+05
- long_name :
- Transducer frequency
- units :
- Hz
- valid_min :
- 0.0
array([ 38000., 120000., 200000.])
- range(range)float640.0 0.2 0.4 ... 204.0 204.2 204.4
array([0.000e+00, 2.000e-01, 4.000e-01, ..., 2.040e+02, 2.042e+02, 2.044e+02])
- Sv(frequency, ping_time, range)float6410.29 2.976 ... -53.82 -53.92
- binning_mode :
- physical units
- range_meter_interval :
- 0.2m
- ping_time_interval :
- 10S
array([[[ 10.2941659 , 2.97588737, -43.93871871, ..., -59.39359035, -58.51848588, -58.15757018], [ 10.2924061 , 2.97471292, -43.94694519, ..., -59.70829734, -58.9021054 , -58.44367479], [ 10.29346681, 2.97471611, -43.94459318, ..., -61.54676199, -60.72512638, -59.91703551], ..., [ 10.29435805, 2.97706468, -43.90696412, ..., -63.01471977, -63.572483 , -63.70741937], [ 10.29427978, 2.97471611, -43.90578922, ..., -62.17410335, -62.75104509, -61.36281038], [ 10.29550428, 2.97588737, -43.90344039, ..., -62.63971134, -61.22632313, -63.20869066]], [[ 24.96402934, 20.07128925, -56.43306531, ..., -45.08606018, -47.919346 , -44.66253456], [ 24.96190237, 20.06894312, -56.52384906, ..., -47.55576147, -45.8308537 , -43.83874558], [ 24.96154232, 20.06659572, -56.66413745, ..., -43.24753479, -44.13437922, -43.52107297], ... [ 24.96127826, 20.06424705, -56.32515814, ..., -50.37329767, -51.33618461, -53.03317593], [ 24.95757481, 20.06894312, -56.42883271, ..., -49.19252972, -47.9563338 , -47.97585831], [ 24.9637024 , 20.06659572, -56.56767333, ..., -49.97089074, -49.75038729, -50.79228821]], [[-103.58906545, -36.25821127, -50.2027861 , ..., -51.4190373 , -48.5644235 , -47.93171624], [-103.56415992, -36.28851311, -50.55051137, ..., -47.9346857 , -49.66127168, -49.83431134], [-103.51259763, -36.2755931 , -50.51168674, ..., -48.64609848, -51.05259416, -49.12039111], ..., [-103.58281502, -36.25557819, -50.49363741, ..., -52.20313571, -53.51419853, -54.62680545], [-103.49834208, -36.27084369, -50.3937374 , ..., -52.81726344, -54.54043897, -53.22778384], [-103.56570192, -36.25681361, -50.54373408, ..., -55.12746209, -53.81929257, -53.91717465]]])
- binning_mode :
- physical units
- range_meter_interval :
- 0.2m
- ping_time_interval :
- 10S
3.3.7. Visualize MVBS interactively using hvPlot¶
To visualize, invert the range axis since the echosounder is upward-looking from a platform at approximately 200 m water depth.
ds_MVBS = ds_MVBS.assign_coords(depth=("range", ds_MVBS["range"].values[::-1]))
ds_MVBS = ds_MVBS.swap_dims({'range': 'depth'}) # set depth as data dimension
ds_MVBS["Sv"].sel(frequency=200000).hvplot.image(
x='ping_time', y='depth',
color='Sv', rasterize=True,
cmap='jet', clim=(-80, -30),
xlabel='Time (UTC)',
ylabel='Depth (m)'
).options(width=800, invert_yaxis=True)
OMP: Info #271: omp_set_nested routine deprecated, please use omp_set_max_active_levels instead.
Note that the reflection from the sea surface shows up at a location below the depth of 0 m. This is because we have not corrected for the actual depth of the platform on which the echosounder is mounted, and the actual sound speed at the time of data collection (which is related to the calculated range) could also be different from the user-defined sound speed stored in the data file. More accurate platform depth information can be obtained using data from the CTD collocated on the moored platform.
3.4. Obtain solar radiation data from an OOI Thredds server¶
Now we have the sonar data ready, the next step is to pull solar radiation data collected by a nearby surface mooring also maintained by the OOI. The Bulk Meteorology Instrument Package is located on the Coastal Endurance Oregon Offshore Surface Mooring.
Note: an earlier version of this notebook used the same dataset but pulled from the National Data Buoy Center (NDBC). We thank the Rutgers OOI Data Lab for pointing out the direct data source in one of the data nuggets.
metbk_url = (
"http://thredds.dataexplorer.oceanobservatories.org/thredds/dodsC/ooigoldcopy/public/"
"CE04OSSM-SBD11-06-METBKA000-recovered_host-metbk_a_dcl_instrument_recovered/"
"deployment0004_CE04OSSM-SBD11-06-METBKA000-recovered_host-metbk_a_dcl_instrument_recovered_20170421T022518.003000-20171013T154805.602000.nc#fillmismatch"
)
metbk_ds = (
xr.open_dataset(metbk_url)
.swap_dims({'obs': 'time'})
.drop('obs')
.sel(time=slice(start_datetime, end_datetime))[['shortwave_irradiance']]
)
metbk_ds.time.attrs.update({'long_name': 'Time', 'units': 'UTC'})
metbk_ds
<xarray.Dataset>
Dimensions: (time: 1441)
Coordinates:
* time (time) datetime64[ns] 2017-08-21T07:00:08.232999936...
Data variables:
shortwave_irradiance (time) float32 2.2 2.3 2.2 2.2 2.2 ... 2.4 2.3 2.3 2.4
Attributes: (12/73)
node: SBD11
comment:
publisher_email:
sourceUrl: http://oceanobservatories.org/
collection_method: recovered_host
stream: metbk_a_dcl_instrument_recovered
... ...
geospatial_vertical_positive: down
lat: 44.36555
lon: -124.9407
DODS.strlen: 14
DODS.dimName: string14
DODS_EXTRA.Unlimited_Dimension: obsxarray.Dataset
- time: 1441
- time(time)datetime64[ns]2017-08-21T07:00:08.232999936 .....
- axis :
- T
- standard_name :
- time
- long_name :
- Time
- _ChunkSizes :
- 10000
- units :
- UTC
array(['2017-08-21T07:00:08.232999936', '2017-08-21T07:01:12.539999744', '2017-08-21T07:02:16.888000000', ..., '2017-08-22T06:58:15.237000192', '2017-08-22T06:59:19.614000128', '2017-08-22T06:59:53.616999936'], dtype='datetime64[ns]')
- shortwave_irradiance(time)float32...
- long_name :
- Downwelling Shortwave Irradiance
- precision :
- 1
- coordinates :
- time lat lon
- data_product_identifier :
- SHRTIRR_L1
- standard_name :
- downwelling_shortwave_flux_in_air
- units :
- W m-2
- _ChunkSizes :
- 10000
array([2.2, 2.3, 2.2, ..., 2.3, 2.3, 2.4], dtype=float32)
- node :
- SBD11
- comment :
- publisher_email :
- sourceUrl :
- http://oceanobservatories.org/
- collection_method :
- recovered_host
- stream :
- metbk_a_dcl_instrument_recovered
- featureType :
- point
- creator_email :
- publisher_name :
- Ocean Observatories Initiative
- date_modified :
- 2021-07-24T08:34:41.513256
- keywords :
- cdm_data_type :
- Point
- references :
- More information can be found at http://oceanobservatories.org/
- Metadata_Conventions :
- Unidata Dataset Discovery v1.0
- date_created :
- 2021-07-24T08:34:41.513253
- id :
- CE04OSSM-SBD11-06-METBKA000-recovered_host-metbk_a_dcl_instrument_recovered
- requestUUID :
- 78426954-0e13-41ea-afe5-f6518108f0ba
- contributor_role :
- summary :
- Dataset Generated by Stream Engine from Ocean Observatories Initiative
- keywords_vocabulary :
- institution :
- Ocean Observatories Initiative
- naming_authority :
- org.oceanobservatories
- feature_Type :
- point
- infoUrl :
- http://oceanobservatories.org/
- license :
- contributor_name :
- uuid :
- 78426954-0e13-41ea-afe5-f6518108f0ba
- creator_name :
- Ocean Observatories Initiative
- title :
- Data produced by Stream Engine version 1.18.0 for CE04OSSM-SBD11-06-METBKA000-recovered_host-metbk_a_dcl_instrument_recovered
- sensor :
- 06-METBKA000
- standard_name_vocabulary :
- NetCDF Climate and Forecast (CF) Metadata Convention Standard Name Table 29
- acknowledgement :
- Conventions :
- CF-1.6
- project :
- Ocean Observatories Initiative
- source :
- CE04OSSM-SBD11-06-METBKA000-recovered_host-metbk_a_dcl_instrument_recovered
- publisher_url :
- http://oceanobservatories.org/
- creator_url :
- http://oceanobservatories.org/
- nodc_template_version :
- NODC_NetCDF_TimeSeries_Orthogonal_Template_v1.1
- subsite :
- CE04OSSM
- processing_level :
- L2
- history :
- 2021-07-24T08:34:41.513224 generated from Stream Engine
- Manufacturer :
- Star Engineering
- ModelNumber :
- ASIMET
- SerialNumber :
- LGR032
- Description :
- Bulk Meteorology Instrument Package: METBK LGR Module
- FirmwareVersion :
- Not specified.
- SoftwareVersion :
- Not specified.
- AssetUniqueID :
- CGINS-METLGR-00032
- Notes :
- Not specified.
- Owner :
- Oregon State University
- RemoteResources :
- []
- ShelfLifeExpirationDate :
- Not specified.
- Mobile :
- False
- AssetManagementRecordLastModified :
- 2021-07-23T11:39:05.965000
- time_coverage_start :
- 2017-04-21T02:25:18.003000
- time_coverage_end :
- 2017-10-13T15:48:05.602000
- time_coverage_resolution :
- P60.00S
- geospatial_lat_min :
- 44.36555
- geospatial_lat_max :
- 44.36555
- geospatial_lat_units :
- degrees_north
- geospatial_lat_resolution :
- 0.1
- geospatial_lon_min :
- -124.9407
- geospatial_lon_max :
- -124.9407
- geospatial_lon_units :
- degrees_east
- geospatial_lon_resolution :
- 0.1
- geospatial_vertical_units :
- meters
- geospatial_vertical_resolution :
- 0.1
- geospatial_vertical_positive :
- down
- lat :
- 44.36555
- lon :
- -124.9407
- DODS.strlen :
- 14
- DODS.dimName :
- string14
- DODS_EXTRA.Unlimited_Dimension :
- obs
3.5. Combine sonar observation with solar radiation measurements¶
We can finally put everything together and figure out the impact of the eclipse-driven reduction in sunlight on marine zooplankton!
metbk_plot = metbk_ds.hvplot.line(
x='time', y='shortwave_irradiance'
).options(width=800, height=200, logy=True, xlim=(start_datetime, end_datetime))
mvbs_plot = ds_MVBS["Sv"].sel(frequency=200000).hvplot.image(
x='ping_time', y='depth',
color='Sv', rasterize=True,
cmap='jet', clim=(-80, -30),
xlabel='Time (UTC)',
ylabel='Depth (m)'
).options(width=800, invert_yaxis=True, xlim=(start_datetime, end_datetime))
(metbk_plot + mvbs_plot).cols(1)
Look how the dip at solar radiation reading matches exactly with the upwarding moving “blip” at UTC 17:21, August 22, 2017 (local time 10:22 AM). During the solar eclipse, the animals were fooled by the temporary mask of the sun and thought it’s getting dark as at dusk!
3.6. Package versions¶
print(f"echopype: {ep.__version__}, xarray: {xr.__version__}, fsspec: {fsspec.__version__}, hvplot: {hvplot.__version__}")
echopype: 0.5.4, xarray: 0.19.0, fsspec: 2021.07.0, hvplot: 0.7.3
import datetime
print(f"{datetime.datetime.utcnow()} +00:00")
2021-10-28 03:08:13.301980 +00:00